Methods and apparatus of video coding for triangle prediction

ABSTRACT

A method for video coding is provided. The method includes: partitioning video pictures into a plurality of coding units (CUs), at least one of which is further portioned into two prediction units (PUs) including at least one triangular shaped PU with a partitioning orientation in one of: from top-left corner to bottom-right corner, and from top-right corner to bottom-left corner; constructing a uni-prediction motion vector candidate list; determining whether a current CU is coded as triangle prediction mode according to coded information; signaling a partition orientation flag indicating the partitioning orientation; and signaling index values that indicate selected entries in the constructed uni-prediction motion vector candidate list.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of International ApplicationPCT/US2019/068975, filed on Dec. 30, 2019, which claims priority to U.S.Provisional Application No. 62/786,545, entitled “Triangle PredictionFlag Signalling in Video Coding” filed on Dec. 30, 2018, and U.S.Provisional Application No. 62/787,219, entitled “Triangle PredictionSyntax Coding” filed on Dec. 31, 2018, all of which are incorporated byreference in their entireties for all purpose.

FIELD

The present application generally relates to video coding andcompression, and in particular but not limited to, methods and apparatusfor motion compensated prediction using triangular prediction unit invideo coding.

BACKGROUND

The following abbreviations and acronyms are herewith defined, at leastsome of which are referred to within the following description:

International Telecommunication Union (ITU), ITU TelecommunicationStandardization Sector (ITU-T), International Organization forStandardization (ISO/IEC), International Electrotechnical Commission(IEC), Moving Picture Experts Group (MPEG), Advanced Video Coding (AVC),High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), JointExploration Test Model (JEM), VVC Test Model (VTM), Joint Video ExpertsTeam (JVET), Video Coding Experts Group (VCEG), Motion Vector (MV),Motion Vector Prediction (MVP), Motion Vector Difference (MVD), MotionVector Field (MVF), Advanced Motion Vector Prediction (AMVP), MotionVector Competition (MVC), Temporal Motion Vector Prediction (TMVP),Control Point Motion Vector (CPMV), Adaptive Loop Filter (ALF),Bi-predictive (B), Block Copy (BC), Context-based Adaptive BinaryArithmetic Coding (CABAC), Context Adaptive Variable Length Coding(CAVLC), Coding Block (CB), encoder/decoder (CODEC), Coded PictureBuffer (CPB), Coding Tree Block (CTB), Coding Tree Unit (CTU), CodingUnit (CU), Discrete Cosine Transform (DCT), Decoded Picture Buffer(DPB), Intra (I), Intra Block Copy (IBC), Predictive (P), PredictionBlock (PB), Probability Interval Partitioning Entropy (PIPE), PictureOrder Count (POC), Picture Unit (PU), Sum Of Absolute Difference (SAD),Sample Adaptive Offset (SAO), Syntax-Based Context-Adaptive BinaryArithmetic Coding (SBAC), Sequence Parameter Set (SPS), Sum Of SquareDifference (SSD), Transform Unit (TU), Quadtree Plus Binary Tree (QTBT),pixel, or picture element (pel), Advanced Temporal Level Motion VectorPrediction (ATMVP), Ultimate Motion Vector Expression (UMVE), Merge Modewith Motion Vector Difference (MMVD), Index (IDX), Combined Inter andIntra Prediction (CIIP), Most Probable Mode (MPM).

Digital video is supported by a variety of electronic devices, such asdigital televisions, laptop or desktop computers, tablet computers,digital cameras, digital recording devices, digital media players, videogaming consoles, smart phones, video teleconferencing devices, videostreaming devices, etc. The electronic devices transmit, receive,encode, decode, and/or store digital video data by implementing videocompression/decompression. Digital video devices implement video codingtechniques, such as those described in the standards defined byVersatile Video Coding (VVC), Joint Exploration Test Model (JEM),MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264/MPEG-4, Part 10, Advanced VideoCoding (AVC), ITU-T H.265/High Efficiency Video Coding (HEVC), andextensions of such standards.

Video coding generally utilizes prediction methods (e.g.,inter-prediction, intra-prediction) that take advantage of redundancypresent in video images or sequences. An important goal of video codingtechniques is to compress video data into a form that uses a lower bitrate, while avoiding or minimizing degradations to video quality. Withever-evolving video services becoming available, encoding techniqueswith better coding efficiency are needed.

Video compression typically includes performing spatial (intra frame)prediction and/or temporal (inter frame) prediction to reduce or removeredundancy inherent in the video data. For block-based video coding, avideo frame is partitioned into one or more slices, each slice havingmultiple video blocks, which may also be referred to as coding treeunits (CTUs). Each CTU may contain one coding unit (CU) or recursivelysplit into smaller CUs until the predefined minimum CU size is reached.Each CU (also named leaf CU) contains one or multiple transform units(TUs) and each CU also contains one or multiple prediction units (PUs).Each CU can be coded in intra, inter or IBC modes. Video blocks in anintra coded (I) slice of a video frame are encoded using spatialprediction with respect to reference samples in neighbor blocks withinthe same video frame. Video blocks in an inter coded (P or B) slice of avideo frame may use spatial prediction with respect to reference samplesin neighbor blocks within the same video frame or temporal predictionwith respect to reference samples in other previous and/or futurereference video frames.

Spatial or temporal prediction based on a reference block that has beenpreviously encoded, e.g., a neighbor block, results in a predictiveblock for a current video block to be coded. The process of finding thereference block may be accomplished by block matching algorithm.Residual data representing pixel differences between the current blockto be coded and the predictive block is referred to as a residual blockor prediction errors. An inter-coded block is encoded according to amotion vector that points to a reference block in a reference frameforming the predictive block, and the residual block. The process ofdetermining the motion vector is typically referred to as motionestimation. An intra coded block is encoded according to an intraprediction mode and the residual block. For further compression, theresidual block is transformed from the pixel domain to a transformdomain, e.g., frequency domain, resulting in residual transformcoefficients, which may then be quantized. The quantized transformcoefficients, initially arranged in a two-dimensional array, may bescanned to produce a one-dimensional vector of transform coefficients,and then entropy encoded into a video bitstream to achieve even morecompression.

The encoded video bitstream is then saved in a computer-readable storagemedium (e.g., flash memory) to be accessed by another electronic devicewith digital video capability or directly transmitted to the electronicdevice wired or wirelessly. The electronic device then performs videodecompression (which is an opposite process to the video compressiondescribed above) by, e.g., parsing the encoded video bitstream to obtainsyntax elements from the bitstream and reconstructing the digital videodata to its original format from the encoded video bitstream based atleast in part on the syntax elements obtained from the bitstream, andrenders the reconstructed digital video data on a display of theelectronic device.

With digital video quality going from high definition, to 4K×2K or even8K×4K, the amount of vide data to be encoded/decoded growsexponentially. It is a constant challenge in terms of how the video datacan be encoded/decoded more efficiently while maintaining the imagequality of the decoded video data.

In a Joint Video Experts Team (JVET) meeting, JVET defined the firstdraft of Versatile Video Coding (VVC) and the VVC Test Model 1 (VTM1)encoding method. It was decided to include a quadtree with nestedmulti-type tree using binary and ternary splits coding block structureas the initial new coding feature of VVC. Since then, the referencesoftware VTM to implement the encoding method and the draft VVC decodingprocess has been developed during the JVET meetings.

SUMMARY

In general, this disclosure describes examples of techniques relating tomotion compensated prediction using triangular prediction unit in videocoding.

According to a first aspect of the present disclosure, there is provideda method for video coding, including: partitioning video pictures into aplurality of coding units (CUs), at least one of which is furtherportioned into two prediction units (PUs) including at least onetriangular shaped PU with a partitioning orientation in one of: fromtop-left corner to bottom-right corner, and from top-right corner tobottom-left corner; constructing a uni-prediction motion vectorcandidate list; determining whether a current CU is coded as triangleprediction mode according to coded information; signaling a partitionorientation flag indicating the partitioning orientation; and signalingindex values that indicate selected entries in the constructeduni-prediction motion vector candidate list.

According to a second aspect of the present disclosure, there isprovided a method for video coding, including: partitioning videopictures into a plurality of coding units (CUs), at least one of whichis further portioned into two prediction units (PUs) including at leastone triangular shaped PU; determining whether a triangle predictionflag, which indicates triangle prediction mode, is to be decoded for acurrent CU; and deriving the triangle prediction flag using CABAC upondetermining that the triangle prediction flag is to be decoded for thecurrent CU; where contexts of the CABAC for the triangle prediction flagare derived based on at least one selected from a group consisting of: aCU area size; a CU shape; and a common context.

According to a third aspect of the present disclosure, there is providedan apparatus for video coding, including: a processor; and a memoryconfigured to store instructions executable by the processor; where theprocessor, upon execution of the instructions, is configured to:partition video pictures into a plurality of coding units (CUs), atleast one of which is further portioned into two prediction units (PUs)including at least one triangular shaped PU with a partitioningorientation in one of: from top-left corner to bottom-right corner, andfrom top-right corner to bottom-left corner; construct a uni-predictionmotion vector candidate list; determine whether a current CU is coded astriangle prediction mode according to coded information; signal apartition orientation flag indicating the partitioning orientation; andsignal index values that indicate selected entries in the constructeduni-prediction motion vector candidate list.

According to a fourth aspect of the present disclosure, there isprovided an apparatus for video coding, including: a processor; and amemory configured to store instructions executable by the processor;where the processor, upon execution of the instructions, is configuredto: partition video pictures into a plurality of coding units (CUs), atleast one of which is further portioned into two prediction units (PUs)including at least one triangular shaped PU; determine whether atriangle prediction flag, which indicates triangle prediction mode, isto be decoded for a current CU; and derive the triangle prediction flagusing CABAC upon determining that the triangle prediction flag is to bedecoded for the current CU; where contexts of the CABAC for the triangleprediction flag are derived based on at least one selected from a groupconsisting of: a CU area size; a CU shape; and a common context.

BRIEF DESCRIPTION OF DRAWINGS

A more particular description of the examples of the present disclosurewill be rendered by reference to specific examples illustrated in theappended drawings. Given that these drawings depict only some examplesand are not therefore considered to be limiting in scope, the exampleswill be described and explained with additional specificity and detailsthrough the use of the accompanying drawings.

FIG. 1 is a block diagram illustrating an exemplary video encoding anddecoding system in accordance with some implementations of the presentdisclosure.

FIG. 2 is a block diagram illustrating an exemplary video encoder inaccordance with some implementations of the present disclosure.

FIG. 3 is a block diagram illustrating an exemplary video decoder inaccordance with some implementations of the present disclosure.

FIG. 4 is a schematic diagram illustrating a QTBT structure inaccordance with some implementations of the present disclosure.

FIG. 5 is a schematic diagram illustrating an example of a picturedivided into CTUs in accordance with some implementations of the presentdisclosure.

FIG. 6 is a schematic diagram illustrating multi-type tree splittingmodes in accordance with some implementations of the present disclosure.

FIG. 7 is a schematic diagram illustrating splitting a CU intotriangular prediction units in accordance with some implementations ofthe present disclosure.

FIG. 8 is a schematic diagram illustrating positions of neighboringblocks in accordance with some implementations of the presentdisclosure.

FIG. 9 is a schematic diagram illustrating an example of weighting witha weighting factor set in accordance with some implementations of thepresent disclosure.

FIG. 10 is a schematic diagram illustrating context derivation forcoding a triangle prediction flag in accordance with someimplementations of the present disclosure.

FIG. 11 is a schematic diagram illustrating examples of PU partitions inaccordance with some implementations of the present disclosure.

FIG. 12 is a schematic diagram illustrating positions of spatial mergecandidates in accordance with some implementations of the presentdisclosure.

FIG. 13 is a schematic diagram illustrating motion vector scaling for atemporal merge candidate in accordance with some implementations of thepresent disclosure.

FIG. 14 is a schematic diagram illustrating candidate position for atemporal merge candidate in accordance with some implementations of thepresent disclosure.

FIG. 15 is a block diagram illustrating an exemplary apparatus for videocoding in accordance with some implementations of the presentdisclosure.

FIG. 16 is a flowchart illustrating an exemplary process of video codingfor motion compensated prediction using triangular prediction inaccordance with some implementations of the present disclosure.

FIG. 17 is a flowchart illustrating an exemplary process of video codingfor triangle prediction flag in accordance with some implementations ofthe present disclosure.

DETAILED DESCRIPTION

Reference will now be made in detail to specific implementations,examples of which are illustrated in the accompanying drawings. In thefollowing detailed description, numerous non-limiting specific detailsare set forth in order to assist in understanding the subject matterpresented herein. But it will be apparent to one of ordinary skill inthe art that various alternatives may be used. For example, it will beapparent to one of ordinary skill in the art that the subject matterpresented herein can be implemented on many types of electronic deviceswith digital video capabilities.

The description of elements in each figure may refer to elements ofother figures. Like-numbers may refer to like-elements in the figures,including alternative embodiments of like-elements.

Reference throughout this specification to “one embodiment,” “anembodiment,” “an example,” “some embodiments,” “some examples,” orsimilar language means that a particular feature, structure, orcharacteristic described is included in at least one embodiment orexample. Thus, instances of the phrases “in one embodiment,” “in anexample,” “in some embodiments,” and similar language throughout thisspecification may, but do not necessarily, all refer to the sameembodiment(s). It may or may not include all the embodiments disclosed.Features, structures, elements, or characteristics described inconnection with one or some embodiments are also applicable to otherembodiments, unless expressly specified otherwise.

The schematic flowchart diagrams and/or schematic block diagrams in thefigures illustrate the architecture, functionality, and operation ofpossible implementations of different apparatuses, systems, methods, andprogram products according to various embodiments. In this regard, eachblock in the schematic flowchart diagrams and/or schematic blockdiagrams may represent a module, segment, or portion of code, whichincludes one or more executable instructions of the code forimplementing the specified logical function(s). One skilled in therelevant art will recognize, however, that the flowchart diagrams neednot necessarily be practiced in the sequence shown and are able to bepracticed without one or more of the specific steps, or with other stepsnot shown.

It should also be noted that, in some alternative implementations, thefunctions noted in the identified blocks may occur out of the ordernoted in the figures. For example, two blocks shown in succession may,in fact, be substantially executed in concurrence, or the blocks maysometimes be executed in reverse order, depending upon the functionalityinvolved. Other steps and methods may be conceived that are equivalentin function, logic, or effect to one or more blocks, or portionsthereof, to the illustrated figures.

The terminology used in the present disclosure is for the purpose ofdescribing particular examples only and is not intended to limit thepresent disclosure. The terms “including,” “comprising,” “having,” andvariations thereof mean “including but not limited to,” unless expresslyspecified otherwise.

It shall be further understood that these terms specify the presence ofstated features, integers, steps, operations, elements and/orcomponents, but do not preclude the presence or addition of one or moreother features, integers, steps, operations, elements, components and/orcombinations thereof.

An enumerated listing of items does not imply that any or all of theitems are mutually exclusive, unless expressly specified otherwise.

As used in this disclosure and the appended claims, the singular forms“a,” “an,” and “the” are intended to include the plural forms as well,and should be interpreted as equivalent to “one or more” or “at leastone,” unless the context clearly indicates otherwise.

It should be understood that the term “and/or” as used herein refers toand includes any and all possible combinations of one or more of theassociated listed items. For example, “A and/or B” may refer to any oneof the following three combinations: existence of A only, existence of Bonly, and co-existence of both A and B.

The character “/” generally indicates an “or” relationship of theassociated items, but may also include an “and” relationship of theassociated items. For example, “A/B” may also include the co-existenceof both A and B, unless the context indicates otherwise.

Throughout the disclosure, the terms “first,” “second,” “third,” andetc. are all used as nomenclature only for references to relevantelements, e.g. devices, components, compositions, steps, and etc.,without implying any spatial or chronological orders, unless expresslyspecified otherwise. For example, a “first device” and a “second device”may refer to two separately formed devices, or two parts, components oroperational states of a same device, and may be named arbitrarily.

A first element and a second element may exist independently. Forexample, some embodiments may include a second element only, without anyfirst element. Accordingly, a second element may be described, prior todescription of a first element, or without description of the firstelement. For example, a “first step” of a method or process may becarried or performed after, or simultaneously with, a “second step.”

As used herein, the term “if” or “when” may be understood to mean “upon”or “in response to” depending on the context. These terms, if appear ina claim, may not indicate that the relevant limitations or features areconditional or optional. For example, a method may comprise steps of: i)when or if condition X is present, function or action X′ is performed,and ii) when or if condition Y is present, function or action Y′ isperformed. It may require the method to be implemented with both thecapability of performing function or action X′, and the capability ofperforming function or action Y′, while the functions X′ and Y′ may bothbe performed, at different times, on multiple executions of the method.It may further be implemented with the capability of detecting orevaluating satisfaction of condition X, and the capability of detectingor evaluating satisfaction of condition Y.

The terms “module,” “sub-module,” “circuit,” “sub-circuit,” “circuitry,”“sub-circuitry,” “unit,” or “sub-unit” may include memory (shared,dedicated, or group) that stores code or instructions that can beexecuted by one or more processors. A module may include one or morecircuits with or without stored code or instructions. The module orcircuit may include one or more components that are directly orindirectly connected. These components may or may not be physicallyattached to, or located adjacent to, one another.

A unit or module may be implemented purely by software, purely byhardware, or by a combination of hardware and software. In a puresoftware implementation, for example, the unit or module may includefunctionally related code blocks or software components, that aredirectly or indirectly linked together, so as to perform a particularfunction.

FIG. 1 is a block diagram illustrating an exemplary system 10 forencoding and decoding video blocks in accordance with someimplementations of the present disclosure. As shown in FIG. 1 , thesystem 10 includes a source device 12 that generates and encodes videodata which is to be decoded at a later time by a destination device 14.The source device 12 and the destination device 14 may be any of a widevariety of electronic devices, including desktop or laptop computers,tablet computers, smart phones, set-top boxes, digital televisions,cameras, display devices, digital media players, video gaming consoles,video streaming device, or the like. In some implementations, the sourcedevice 12 and the destination device 14 are equipped with wirelesscommunication capabilities.

In some implementations, the destination device 14 may receive theencoded video data to be decoded via a link 16. The link 16 may be anytype of communication medium or device capable of moving the encodedvideo data from the source device 12 to the destination device 14. Inone example, the link 16 may be a communication medium to enable thesource device 12 to transmit the encoded video data directly to thedestination device 14 in real-time. The encoded video data may bemodulated according to a communication standard, such as a wirelesscommunication protocol, and transmitted to the destination device 14.The communication medium may be any wireless or wired communicationmedium, such as a radio frequency (RF) spectrum or one or more physicaltransmission lines. The communication medium may form part of apacket-based network, such as a local area network, a wide-area network,or a global network such as the Internet. The communication medium mayinclude routers, switches, base stations, or any other equipment thatmay be useful to facilitate communication from the source device 12 tothe destination device 14.

In some other implementations, the encoded video data may be transmittedfrom an output interface 22 to a storage device 32. Subsequently, theencoded video data in the storage device 32 may be accessed by thedestination device 14 via an input interface 28. The storage device 32may include any of a variety of distributed or locally accessed datastorage media such as a hard drive, Blu-ray discs, DVDs, CD-ROMs, flashmemory, volatile or non-volatile memory, or any other suitable digitalstorage media for storing encoded video data. In a further example, thestorage device 32 may correspond to a file server or anotherintermediate storage device that may hold the encoded video datagenerated by the source device 12. The destination device 14 may accessthe stored video data from the storage device 32 via streaming ordownloading. The file server may be any type of computer capable ofstoring encoded video data and transmitting the encoded video data tothe destination device 14. Exemplary file servers include a web server(e.g., for a website), an FTP server, network attached storage (NAS)devices, or a local disk drive. The destination device 14 may access theencoded video data through any standard data connection, including awireless channel (e.g., a Wi-Fi connection), a wired connection (e.g.,DSL, cable modem, etc.), or a combination of both which is suitable foraccessing encoded video data stored on a file server. The transmissionof the encoded video data from the storage device 32 may be a streamingtransmission, a download transmission, or a combination of both.

As shown in FIG. 1 , the source device 12 includes a video source 18, avideo encoder 20 and the output interface 22. The video source 18 mayinclude a source such as a video capture device, e.g., a video camera, avideo archive containing previously captured video, a video feedinterface to receive video from a video content provider, and/or acomputer graphics system for generating computer graphics data as thesource video, or a combination of such sources. As one example, if thevideo source 18 is a video camera of a security surveillance system, thesource device 12 and the destination device 14 may be camera phones orvideo phones. However, the implementations described in the presentdisclosure may be applicable to video coding in general, and may beapplied to wireless and/or wired applications.

The captured, pre-captured, or computer-generated video may be encodedby the video encoder 20. The encoded video data may be transmitteddirectly to the destination device 14 via the output interface 22 of thesource device 12. The encoded video data may also (or alternatively) bestored onto the storage device 32 for later access by the destinationdevice 14 or other devices, for decoding and/or playback. The outputinterface 22 may further include a modem and/or a transmitter.

The destination device 14 includes an input interface 28, a videodecoder 30, and a display device 34. The input interface 28 may includea receiver and/or a modem and receive the encoded video data over thelink 16. The encoded video data communicated over the link 16, orprovided on the storage device 32, may include a variety of syntaxelements generated by the video encoder 20 for use by the video decoder30 in decoding the video data. Such syntax elements may be includedwithin the encoded video data transmitted on a communication medium,stored on a storage medium, or stored a file server.

In some implementations, the destination device 14 may include thedisplay device 34, which may be an integrated display device or anexternal display device that is configured to communicate with thedestination device 14. The display device 34 displays the decoded videodata to a user, and may be any of a variety of display devices such as aliquid crystal display (LCD), a plasma display, an organic lightemitting diode (OLED) display, or another type of display device.

The video encoder 20 and the video decoder 30 may operate according toproprietary or industry standards, such as VVC, HEVC, MPEG-4, Part 10,Advanced Video Coding (AVC), or extensions of such standards. It shouldbe understood that the present disclosure is not limited to a specificvideo coding/decoding standard and may be applicable to other videocoding/decoding standards. It is generally contemplated that the videoencoder 20 of the source device 12 may be configured to encode videodata according to any of these current or future standards. Similarly,it is also generally contemplated that the video decoder 30 of thedestination device 14 may be configured to decode video data accordingto any of these current or future standards.

The video encoder 20 and the video decoder 30 each may be implemented asany of a variety of suitable encoder circuitry, such as one or moremicroprocessors, digital signal processors (DSPs), application specificintegrated circuits (ASICs), field programmable gate arrays (FPGAs),discrete logic, software, hardware, firmware or any combinationsthereof. When implemented partially in software, an electronic devicemay store instructions for the software in a suitable, non-transitorycomputer-readable medium and execute the instructions in hardware usingone or more processors to perform the video coding/decoding operationsdisclosed in the present disclosure. Each of the video encoder 20 andthe video decoder 30 may be included in one or more encoders ordecoders, either of which may be integrated as part of a combinedencoder/decoder (CODEC) in a respective device.

FIG. 2 is a block diagram illustrating an exemplary video encoder 20 inaccordance with some implementations described in the presentdisclosure. The video encoder 20 may perform intra and inter predictivecoding of video blocks within video frames. Intra predictive codingrelies on spatial prediction to reduce or remove spatial redundancy invideo data within a given video frame or picture. Inter predictivecoding relies on temporal prediction to reduce or remove temporalredundancy in video data within adjacent video frames or pictures of avideo sequence.

As shown in FIG. 2 , the video encoder 20 includes a video data memory40, a prediction processing unit 41, a decoded picture buffer (DPB) 64,a summer 50, a transform processing unit 52, a quantization unit 54, andan entropy encoding unit 56. The prediction processing unit 41 furtherincludes a motion estimation unit 42, a motion compensation unit 44, apartition unit 45, an intra prediction processing unit 46, an intrablock copy (IBC) unit 48, and an intra/inter mode decision unit 49. Insome implementations, the video encoder 20 also includes an inversequantization unit 58, an inverse transform processing unit 60, and asummer 62 for video block reconstruction. A deblocking filter may bepositioned between the summer 62 and the DPB 64 to filter blockboundaries to remove blockiness artifacts from a reconstructed video. Anin-loop filter 63 may also be used in addition to the deblocking filterto filter the output of summer 62. The video encoder 20 may take theform of a fixed or programmable hardware unit or may be divided amongone or more of the fixed or programmable hardware units.

The video data memory 40 may store video data to be encoded by thecomponents of the video encoder 20. The video data in the video datamemory 40 may be obtained, for example, from the video source 18. TheDPB 64 is a buffer that stores reference video data for use in encodingvideo data by the video encoder 20 (e.g., in intra or inter predictivecoding modes). The video data memory 40 and DPB 64 may be any of avariety of memory devices. In various examples, the video data memory 40may be on-chip with other components of the video encoder 20, oroff-chip relative to those components.

As shown in FIG. 2 , after receiving video data, the partition unit 45within the prediction processing unit 41 partitions the video data intovideo blocks. This partitioning may also include partitioning a videoframe into slices, tiles, or other larger coding units (CUs) accordingto a predefined splitting structures such as quad-tree structureassociated with the video data. The video frame may be divided intomultiple video blocks (or sets of video blocks referred to as tiles).The prediction processing unit 41 may select one of a plurality ofpossible predictive coding modes, such as one of a plurality of intrapredictive coding modes or one of a plurality of inter predictive codingmodes, for the current video block based on error results (e.g., codingrate and the level of distortion). The prediction processing unit 41 mayprovide the resulting intra or inter prediction coded block to thesummer 50 to generate a residual block and to the summer 62 toreconstruct the encoded block for use as part of a reference framesubsequently. The prediction processing unit 41 also provides syntaxelements, such as motion vectors, intra-mode indicators, partitioninformation, and other such syntax information, to the entropy encodingunit 56.

In order to select an appropriate intra predictive coding mode for thecurrent video block, the intra prediction processing unit 46 within theprediction processing unit 41 may perform intra predictive coding of thecurrent video block relative to one or more neighbor blocks in the sameframe as the current block to be coded to provide spatial prediction.The motion estimation unit 42 and the motion compensation unit 44 withinthe prediction processing unit 41 perform inter predictive coding of thecurrent video block relative to one or more predictive blocks in one ormore reference frames to provide temporal prediction. The video encoder20 may perform multiple coding passes, e.g., to select an appropriatecoding mode for each block of the video data.

In some implementations, the motion estimation unit 42 determines theinter prediction mode for a current video frame by generating a motionvector, which indicates the displacement of a prediction unit (PU) of avideo block within the current video frame relative to a predictiveblock within a reference video frame, according to a predeterminedpattern within a sequence of video frames. Motion estimation, performedby the motion estimation unit 42, is the process of generating motionvectors, which estimate motion for video blocks. A motion vector, forexample, may indicate the displacement of a PU of a video block within acurrent video frame or picture relative to a predictive block within areference frame (or other coded unit) relative to the current blockbeing coded within the current frame (or other coded unit). Thepredetermined pattern may designate video frames in the sequence as Pframes or B frames. The intra BC unit 48 may determine vectors, e.g.,block vectors, for intra BC coding in a manner similar to thedetermination of motion vectors by the motion estimation unit 42 forinter prediction, or may utilize the motion estimation unit 42 todetermine the block vector.

A predictive block is a block of a reference frame that is deemed asclosely matching the PU of the video block to be coded in terms of pixeldifference, which may be determined by sum of absolute difference (SAD),sum of square difference (SSD), or other difference metrics. In someimplementations, the video encoder 20 may calculate values forsub-integer pixel positions of reference frames stored in the DPB 64.For example, the video encoder 20 may interpolate values of one-quarterpixel positions, one-eighth pixel positions, or other fractional pixelpositions of the reference frame. Therefore, the motion estimation unit42 may perform a motion search relative to the full pixel positions andfractional pixel positions, and output a motion vector with fractionalpixel precision.

The motion estimation unit 42 calculates a motion vector for a PU of avideo block in an inter prediction coded frame by comparing the positionof the PU to the position of a predictive block of a reference frameselected from a first reference frame list (e.g., List 0) or a secondreference frame list (e.g., List 1), each of which identifies one ormore reference frames stored in the DPB 64. The motion estimation unit42 sends the calculated motion vector to the motion compensation unit 44and then to the entropy encoding unit 56.

Motion compensation, performed by the motion compensation unit 44, mayinvolve fetching or generating the predictive block based on the motionvector determined by the motion estimation unit 42. Upon receiving themotion vector for the PU of the current video block, the motioncompensation unit 44 may locate a predictive block to which the motionvector points in one of the reference frame lists, retrieve thepredictive block from the DPB 64, and forward the predictive block tothe summer 50. The summer 50 then forms a residual video block of pixeldifference values by subtracting pixel values of the predictive blockprovided by the motion compensation unit 44 from the pixel values of thecurrent video block being coded. The pixel difference values forming theresidual vide block may include luma or chroma difference components orboth. The motion compensation unit 44 may also generate syntax elementsassociated with the video blocks of a video frame for use by the videodecoder 30 in decoding the video blocks of the video frame. The syntaxelements may include, for example, syntax elements defining the motionvector used to identify the predictive block, any flags indicating theprediction mode, or any other syntax information described herein. Notethat the motion estimation unit 42 and the motion compensation unit 44may be highly integrated, but they are illustrated separately forconceptual purposes.

In some implementations, the intra BC unit 48 may generate vectors andfetch predictive blocks in a manner similar to that described above inconnection with the motion estimation unit 42 and the motioncompensation unit 44, but with the predictive blocks being in the sameframe as the current block being coded and with the vectors beingreferred to as block vectors as opposed to motion vectors. Inparticular, the intra BC unit 48 may determine an intra-prediction modeto use to encode a current block. In some examples, the intra BC unit 48may encode a current block using various intra-prediction modes, e.g.,during separate encoding passes, and test their performance throughrate-distortion analysis. Next, the intra BC unit 48 may select, amongthe various tested intra-prediction modes, an appropriateintra-prediction mode to use and generate an intra-mode indicatoraccordingly. For example, the intra BC unit 48 may calculaterate-distortion values using a rate-distortion analysis for the varioustested intra-prediction modes, and select the intra-prediction modehaving the best rate-distortion characteristics among the tested modesas the appropriate intra-prediction mode to use. Rate-distortionanalysis generally determines an amount of distortion (or error) betweenan encoded block and an original, unencoded block that was encoded toproduce the encoded block, as well as a bit rate (i.e., a number ofbits) used to produce the encoded block. The intra BC unit 48 maycalculate ratios from the distortions and rates for the various encodedblocks to determine which intra-prediction mode exhibits the bestrate-distortion value for the block.

In other examples, the intra BC unit 48 may use the motion estimationunit 42 and the motion compensation unit 44, in whole or in part, toperform such functions for intra BC prediction according to theimplementations described herein. In either case, for intra block copy,a predictive block may be a block that is deemed as closely matching theblock to be coded, in terms of pixel difference, which may be determinedby sum of absolute difference (SAD), sum of squared difference (SSD), orother difference metrics, and identification of the predictive block mayinclude calculation of values for sub-integer pixel positions.

Whether the predictive block is from the same frame according to theintra prediction, or a different frame according to the interprediction, the video encoder 20 may form a residual video block bysubtracting pixel values of the predictive block from the pixel valuesof the current video block being coded, forming pixel difference values.The pixel difference values forming the residual video block may includeboth luma and chroma component differences.

The intra prediction processing unit 46 may intra-predict a currentvideo block, as an alternative to the inter-prediction performed by themotion estimation unit 42 and the motion compensation unit 44, or theintra block copy prediction performed by the intra BC unit 48, asdescribed above. In particular, the intra prediction processing unit 46may determine an intra prediction mode to use to encode a current block.To do so, the intra prediction processing unit 46 may encode a currentblock using various intra prediction modes, e.g., during separateencoding passes, and the intra prediction processing unit 46 (or a modeselect unit, in some examples) may select an appropriate intraprediction mode to use from the tested intra prediction modes. The intraprediction processing unit 46 may provide information indicative of theselected intra-prediction mode for the block to the entropy encodingunit 56. The entropy encoding unit 56 may encode the informationindicating the selected intra-prediction mode in the bitstream.

After the prediction processing unit 41 determines the predictive blockfor the current video block via either inter prediction or intraprediction, the summer 50 forms a residual video block by subtractingthe predictive block from the current video block. The residual videodata in the residual block may be included in one or more transformunits (TUs) and is provided to the transform processing unit 52. Thetransform processing unit 52 transforms the residual video data intoresidual transform coefficients using a transform, such as a discretecosine transform (DCT) or a conceptually similar transform.

The transform processing unit 52 may send the resulting transformcoefficients to the quantization unit 54. The quantization unit 54quantizes the transform coefficients to further reduce bit rate. Thequantization process may also reduce the bit depth associated with someor all of the coefficients. The degree of quantization may be modifiedby adjusting a quantization parameter. In some examples, thequantization unit 54 may then perform a scan of a matrix including thequantized transform coefficients. Alternatively, the entropy encodingunit 56 may perform the scan.

Following quantization, the entropy encoding unit 56 entropy encodes thequantized transform coefficients into a video bitstream using, e.g.,context adaptive variable length coding (CAVLC), context adaptive binaryarithmetic coding (CABAC), syntax-based context-adaptive binaryarithmetic coding (SBAC), probability interval partitioning entropy(PIPE) coding or another entropy encoding methodology or technique. Theencoded bitstream may then be transmitted to the video decoder 30, orarchived in the storage device 32 for later transmission to or retrievalby the video decoder 30. The entropy encoding unit 56 may also entropyencode the motion vectors and the other syntax elements for the currentvideo frame being coded.

The inverse quantization unit 58 and the inverse transform processingunit 60 apply inverse quantization and inverse transformation,respectively, to reconstruct the residual video block in the pixeldomain for generating a reference block for prediction of other videoblocks. As noted above, the motion compensation unit 44 may generate amotion compensated predictive block from one or more reference blocks ofthe frames stored in the DPB 64. The motion compensation unit 44 mayalso apply one or more interpolation filters to the predictive block tocalculate sub-integer pixel values for use in motion estimation.

The summer 62 adds the reconstructed residual block to the motioncompensated predictive block produced by the motion compensation unit 44to produce a reference block for storage in the DPB 64. The referenceblock may then be used by the intra BC unit 48, the motion estimationunit 42 and the motion compensation unit 44 as a predictive block tointer predict another video block in a subsequent video frame.

In an example of video coding using the video encoder 20, a video frameis partitioned into blocks for processing. For each given video block, aprediction is formed based on either inter prediction or intraprediction. In inter prediction, predictors or predictive blocks may beformed through motion estimation and motion compensation, based onpixels from previously reconstructed frames. In intra prediction,predictors may be formed based on reconstructed pixels in the currentframe. Through mode decision, a best predictor may be chosen to predicta current block.

The prediction residual or the residual block (i.e. the differencebetween a current block and its predictor) is sent to a transformmodule, for example, the transform processing unit 52. Transformcoefficients are then sent to a quantization module, e.g., thequantization unit 54, for entropy reduction. Quantized coefficients arefed to an entropy coding module, e.g., the entropy encoding unit 56, togenerate compressed video bitstream. As shown in FIG. 2 , predictionrelated information from inter and/or intra prediction modules, such asblock partition information, motion vectors, reference picture index,and intra prediction mode, etc., is also going through an entropy codingmodule, e.g., the entropy encoding unit 56, and then saved into thebitstream.

In the video encoder 20, decoder related modules may be needed in orderto reconstruct pixels for prediction purpose. First, the predictionresidual is reconstructed through inverse quantization and inversetransform. The reconstructed prediction residual is then combined withthe predictor to generate unfiltered reconstructed pixels for a currentblock.

To improve coding efficiency and visual quality, an in-loop filter 63 iscommonly used. For example, a deblocking filter is available in AVC,HEVC as well as VVC. In HEVC, an additional in-loop filter called SAO(sample adaptive offset) may be defined to further improve codingefficiency. In VVC, an in-loop filter 63 called ALF (adaptive loopfilter) may be employed.

These in-loop filter operations are optional. Turning on the in-loopfilters usually helps improve coding efficiency and visual quality. Theymay also be turned-off as an encoder decision to save computationcomplexity.

It should be noted that intra prediction is usually based on unfilteredreconstructed pixels, while inter prediction is based on filteredreconstructed pixels if these filters are turned on by the encoder.

FIG. 3 is a block diagram illustrating an exemplary video decoder 30 inaccordance with some implementations of the present disclosure. Thevideo decoder 30 includes a video data memory 79, an entropy decodingunit 80, a prediction processing unit 81, an inverse quantization unit86, an inverse transform processing unit 88, a summer 90, and a DPB 92.The prediction processing unit 81 further includes a motion compensationunit 82, an intra prediction unit 84, an intra BC unit 85, and anintra/inter mode selection unit 87. The video decoder 30 may perform adecoding process generally reciprocal to the encoding process describedabove with respect to the video encoder 20 in connection with FIG. 2 .For example, the motion compensation unit 82 may generate predictiondata based on motion vectors received from the entropy decoding unit 80,while the intra prediction unit 84 may generate prediction data based onintra-prediction mode indicators received from the entropy decoding unit80.

In an example of video decoding using the video decoder 30, thebitstream received is decoded through the entropy decoding unit 80 toderive quantized coefficient levels (or quantized coefficients) andprediction related information. Quantized coefficient levels are thenprocessed through the inverse quantization unit 86 and the inversetransform processing unit 88 to obtain reconstructed residual block. Apredictor or predictive block is formed through either intra predictionor motion compensation process based on prediction related informationdecoded. The unfiltered reconstructed pixels are obtained by summing upthe reconstructed residual block and the predictor. In the case that anin-loop filter is turned on, filtering operations are performed on thesepixels to derive the final reconstructed video for output.

In some examples, a unit of the video decoder 30 may be tasked toperform the implementations of the present disclosure. Also, in someexamples, the implementations of the present disclosure may be dividedamong one or more of the units of the video decoder 30. For example, theintra BC unit 85 may perform the implementations of the presentdisclosure, alone, or in combination with other units of the videodecoder 30, such as the motion compensation unit 82, the intraprediction unit 84, and the entropy decoding unit 80. In some examples,the video decoder 30 may not include the intra BC unit 85 and thefunctionality of the intra BC unit 85 may be performed by othercomponents of the prediction processing unit 81, such as the motioncompensation unit 82.

The video data memory 79 may store video data, such as an encoded videobitstream, to be decoded by the other components of the video decoder30. The video data stored in the video data memory 79 may be obtained,for example, from the storage device 32, from a local video source, suchas a camera, via wired or wireless network communication of video data,or by accessing physical data storage media (e.g., a flash drive or harddisk). The video data memory 79 may include a coded picture buffer (CPB)that stores encoded video data from an encoded video bitstream. Thedecoded picture buffer (DPB) 92 of video decoder 30 stores referencevideo data for use in decoding video data by the video decoder 30 (e.g.,in intra or inter predictive coding modes). The video data memory 79 andthe DPB 92 may be formed by any of a variety of memory devices, such asdynamic random access memory (DRAM), including synchronous DRAM (SDRAM),magneto-resistive RAM (MRAM), resistive RAM (RRAM), or other types ofmemory devices. For illustrative purpose, the video data memory 79 andthe DPB 92 are depicted as two distinct components of the video decoder30 in FIG. 3 . But it will be apparent to one skilled in the art thatthe video data memory 79 and the DPB 92 may be provided by the samememory device or separate memory devices. In some examples, the videodata memory 79 may be on-chip with other components of the video decoder30, or off-chip relative to those components.

During the decoding process, the video decoder 30 receives an encodedvideo bitstream that represents video blocks of an encoded video frameand associated syntax elements. The video decoder 30 may receive thesyntax elements at the video frame level and/or the video block level.The entropy decoding unit 80 of the video decoder 30 entropy decodes thebitstream to generate quantized coefficients, motion vectors orintra-prediction mode indicators, and other syntax elements. The entropydecoding unit 80 then forwards the motion vectors and other syntaxelements to the prediction processing unit 81.

When the video frame is coded as an intra predictive coded (I) frame orfor intra coded predictive blocks in other types of frames, the intraprediction unit 84 of the prediction processing unit 81 may generateprediction data for a video block of the current video frame based on asignaled intra prediction mode and reference data from previouslydecoded blocks of the current frame.

When the video frame is coded as an inter-predictive coded (i.e., B orP) frame, the motion compensation unit 82 of the prediction processingunit 81 produces one or more predictive blocks for a video block of thecurrent video frame based on the motion vectors and other syntaxelements received from the entropy decoding unit 80. Each of thepredictive blocks may be produced from a reference frame within one ofthe reference frame lists. The video decoder 30 may construct thereference frame lists, e.g., List0 and List1, using default constructiontechniques based on reference frames stored in the DPB 92.

In some examples, when the video block is coded according to the intraBC mode described herein, the intra BC unit 85 of the predictionprocessing unit 81 produces predictive blocks for the current videoblock based on block vectors and other syntax elements received from theentropy decoding unit 80. The predictive blocks may be within areconstructed region of the same picture as the current video blockdefined by the video encoder 20.

The motion compensation unit 82 and/or the intra BC unit 85 determinesprediction information for a video block of the current video frame byparsing the motion vectors and other syntax elements, and then uses theprediction information to produce the predictive blocks for the currentvideo block being decoded. For example, the motion compensation unit 82uses some of the received syntax elements to determine a prediction mode(e.g., intra or inter prediction) used to code video blocks of the videoframe, an inter prediction frame type (e.g., B or P), constructioninformation for one or more of the reference frame lists for the frame,motion vectors for each inter predictive encoded video block of theframe, inter prediction status for each inter predictive coded videoblock of the frame, and other information to decode the video blocks inthe current video frame.

Similarly, the intra BC unit 85 may use some of the received syntaxelements, e.g., a flag, to determine that the current video block waspredicted using the intra BC mode, construction information of whichvideo blocks of the frame are within the reconstructed region and shouldbe stored in the DPB 92, block vectors for each intra BC predicted videoblock of the frame, intra BC prediction status for each intra BCpredicted video block of the frame, and other information to decode thevideo blocks in the current video frame.

The motion compensation unit 82 may also perform interpolation using theinterpolation filters as used by the video encoder 20 during encoding ofthe video blocks to calculate interpolated values for sub-integer pixelsof reference blocks. In this case, the motion compensation unit 82 maydetermine the interpolation filters used by the video encoder 20 fromthe received syntax elements and use the interpolation filters toproduce predictive blocks.

The inverse quantization unit 86 inverse quantizes the quantizedtransform coefficients provided in the bitstream and entropy decoded bythe entropy decoding unit 80 using the same quantization parametercalculated by the video encoder 20 for each video block in the videoframe to determine a degree of quantization. The inverse transformprocessing unit 88 applies an inverse transform, e.g., an inverse DCT,an inverse integer transform, or a conceptually similar inversetransform process, to the transform coefficients in order to reconstructthe residual blocks in the pixel domain.

After the motion compensation unit 82 or the intra BC unit 85 generatesthe predictive block for the current video block based on the vectorsand other syntax elements, the summer 90 reconstructs decoded videoblock for the current video block by summing the residual block from theinverse transform processing unit 88 and a corresponding predictiveblock generated by the motion compensation unit 82 and the intra BC unit85. An in-loop filter 91 may be positioned between the summer 90 and theDPB 92 to further process the decoded video block. The decoded videoblocks in a given frame are then stored in the DPB 92, which storesreference frames used for subsequent motion compensation of next videoblocks. The DPB 92, or a memory device separate from the DPB 92, mayalso store the decoded video for later presentation on a display device,such as display device 34 of FIG. 1 .

In a typical video coding process, a video sequence typically includesan ordered set of frames or pictures. Each frame may include threesample arrays, denoted SL, SCb, and SCr. SL is a two-dimensional arrayof luma samples. SCb is a two-dimensional array of Cb chroma samples.SCr is a two-dimensional array of Cr chroma samples. In other instances,a frame may be monochrome and therefore includes only onetwo-dimensional array of luma samples.

FIG. 4 is a schematic diagram illustrating a quadtree plus binary tree(QTBT) structure in accordance with some implementations of the presentdisclosure.

Video coding/decoding standards mentioned above, such as VVC, JEM, HEVC,MPEG-4, Part 10, are conceptually similar. For example, they all useblock-based processing. Block partitioning schemes in some standards areelaborated below.

HEVC is based on a hybrid block-based motion-compensated transformcoding architecture. The basic unit for compression is termed codingtree unit (CTU). The maximum CTU size is defined as up to 64 by 64 lumapixels, and two blocks of 32 by 32 chroma pixels for 4:2:0 chromaformat. Each CTU may contain one coding unit (CU) or recursively splitinto four smaller CUs until the predefined minimum CU size is reached.Each CU (also named leaf CU) contains one or multiple prediction units(PUs) and a tree of transform units (TUs).

In general, except for monochrome content, a CTU may include one lumacoding tree block (CTB) and two corresponding chroma CTBs; a CU mayinclude one luma coding block (CB) and two corresponding chroma CBs; aPU may include one luma prediction block (PB) and two correspondingchroma PBs; and a TU may include one luma transform block (TB) and twocorresponding chroma TBs. However, exceptions may occur because theminimum TB size is 4×4 for both luma and chroma (i.e., no 2×2 chroma TBis supported for 4:2:0 color format) and each intra chroma CB always hasonly one intra chroma PB regardless of the number of intra luma PBs inthe corresponding intra luma CB.

For an intra CU, the luma CB can be predicted by one or four luma PBs,and each of the two chroma CBs is always predicted by one chroma PB,where each luma PB has one intra luma prediction mode and the two chromaPBs share one intra chroma prediction mode. Moreover, for the intra CU,the TB size cannot be larger than the PB size. In each PB, the intraprediction is applied to predict samples of each TB inside the PB fromneighboring reconstructed samples of the TB. For each PB, in addition to33 directional intra prediction modes, DC and planar modes are alsosupported to predict flat regions and gradually varying regions,respectively.

For each inter PU, one of three prediction modes including inter, skip,and merge, may be selected. Generally speaking, a motion vectorcompetition (MVC) scheme is introduced to select a motion candidate froma given candidate set that includes spatial and temporal motioncandidates. Multiple references to the motion estimation allow findingthe best reference in 2 possible reconstructed reference picture lists(namely List 0 and List 1). For the inter mode (termed AMVP mode, whereAMVP stands for advanced motion vector prediction), inter predictionindicators (List 0, List 1, or bi-directional prediction), referenceindices, motion candidate indices, motion vector differences (MVDs) andprediction residual are transmitted. As for the skip mode and the mergemode, only merge indices are transmitted, and the current PU inheritsthe inter prediction indicator, reference indices, and motion vectorsfrom a neighboring PU referred by the coded merge index. In the case ofa skip coded CU, the residual signal is also omitted.

The Joint Exploration Test Model (JEM) is built up on top of the HEVCtest model. The basic encoding and decoding flowchart of HEVC is keptunchanged in the JEM; however, the design elements of most importantmodules, including the modules of block structure, intra and interprediction, residue transform, loop filter and entropy coding, aresomewhat modified and additional coding tools are added. The followingnew coding features are included in the JEM.

In HEVC, a CTU is split into CUs by using a quadtree structure denotedas coding tree to adapt to various local characteristics. The decisionwhether to code a picture area using inter-picture (temporal) orintra-picture (spatial) prediction is made at the CU level. Each CU canbe further split into one, two or four PUs according to the PU splittingtype. Inside one PU, the same prediction process is applied and therelevant information is transmitted to the decoder on a PU basis. Afterobtaining the residual block by applying the prediction process based onthe PU splitting type, a CU can be partitioned into transform units(TUs) according to another quadtree structure similar to the coding treefor the CU. One of key features of the HEVC structure is that it has themultiple partition conceptions including CU, PU, and TU.

The QTBT structure removes the concepts of multiple partition types,i.e., it removes the separation of the CU, PU and TU concepts, andsupports more flexibility for CU partition shapes. In the QTBT blockstructure, a CU may have either a square or rectangular shape. As shownin FIG. 4 , a coding tree unit (CTU) is first partitioned by aquaternary tree (i.e., quadtree) structure. The quadtree leaf nodes maybe further partitioned by a binary tree structure. There are twosplitting types in the binary tree splitting: symmetric horizontalsplitting and symmetric vertical splitting. The binary tree leaf nodesare called coding units (CUs), and that segmentation is used forprediction and transform processing without any further partitioning.This means that the CU, PU and TU have the same block size in the QTBTcoding block structure. In the JEM, a CU sometimes consists of codingblocks (CBs) of different colour components, e.g., one CU contains oneluma CB and two chroma CBs in the case of P and B slices of the 4:2:0chroma format, and sometimes consists of a CB of a single component,e.g., one CU contains only one luma CB or just two chroma CBs in thecase of I slices.

The following parameters are defined for the QTBT partitioning scheme.

-   -   CTU size: the root node size of a quadtree, the same concept as        in the HEVC;    -   MinQTSize: the minimum allowed quadtree leaf node size;    -   MaxBTSize: the maximum allowed binary tree root node size;    -   MaxBTDepth: the maximum allowed binary tree depth;    -   MinBTSize: the minimum allowed binary tree leaf node size.

In one example of the QTBT partitioning structure, the CTU size is setas 128×128 luma samples with two corresponding 64×64 blocks of chromasamples (with a 4:2:0 chroma format), the MinQTSize is set as 16×16, theMaxBTSize is set as 64×64, the MinBTSize (for both width and height) isset as 4×4, and the MaxBTDepth is set as 4. The quadtree partitioning isapplied to the CTU first to generate quadtree leaf nodes. The quadtreeleaf nodes may have a size from 16×16 (i.e., the MinQTSize) to 128×128(i.e., the CTU size). If the quadtree leaf node is 128×128, it will notbe further split by the binary tree since the size exceeds the MaxBTSize(i.e., 64×64). Otherwise, the quadtree leaf node could be furtherpartitioned by the binary tree. Therefore, the quadtree leaf node isalso the root node for the binary tree and it has the binary tree depthas 0. When the binary tree depth reaches MaxBTDepth (i.e., 4), nofurther splitting is considered. When the binary tree node has a widthequal to MinBTSize (i.e., 4), no further horizontal splitting isconsidered. Similarly, when the binary tree node has a height equal toMinBTSize, no further vertical splitting is considered. The leaf nodesof the binary tree are further processed by prediction and transformprocessing without any further partitioning. In the JEM, the maximum CTUsize is 256×256 luma samples.

FIG. 4 illustrates an example of block partitioning by using the QTBTscheme, and the corresponding tree representation. The solid linesindicate quadtree splitting and dotted lines indicate binary treesplitting. As shown in FIG. 4 , the coding tree unit (CTU) 400 is firstpartitioned by a quadtree structure, and three of the four quadtree leafnodes 402, 404, 406, 408 are further partitioned by either a quadtreestructure or a binary tree structure. For example, the quadtree leafnode 406 is further partitioned by quadtree splitting; the quadtree leafnode 404 is further partitioned into two leaf nodes 404 a, 404 b bybinary tree splitting; and the quadtree leaf node 402 is also furtherpartitioned by binary tree splitting. In each splitting (i.e., non-leaf)node of the binary tree, one flag is signaled to indicate whichsplitting type (i.e., horizontal or vertical) is used, where 0 indicateshorizontal splitting and 1 indicates vertical splitting. For example,for the quadtree leaf node 404, 0 is signaled to indicate horizontalsplitting, and for the quadtree leaf node 402, 1 is signaled to indicatevertical splitting. For quadtree splitting, there is no need to indicatethe splitting type since quadtree splitting always splits a block bothhorizontally and vertically to produce 4 sub-blocks with an equal size.

In addition, the QTBT scheme supports the ability for the luma andchroma to have a separate QTBT structure. Currently, for P and B slices,the luma and chroma CTBs in one CTU share the same QTBT structure.However, for I slices, the luma CTB is partitioned into luma CUs by aQTBT structure, and the chroma CTBs are partitioned into chroma CUs byanother QTBT structure. This means that a CU in an I slice consists of acoding block of the luma component or coding blocks of two chromacomponents, and a CU in a P or B slice consists of coding blocks of allthree colour components.

In a Joint Video Experts Team (JVET) meeting, the JVET defined the firstdraft of the Versatile Video Coding (VVC) and the VVC Test Model 1(VTM1) encoding method. It was decided to include a quadtree with nestedmulti-type tree using binary and ternary splits coding block structureas the initial new coding feature of VVC.

In VVC, the picture partitioning structure divides the input video intoblocks called coding tree units (CTUs). A CTU is split using a quadtreewith nested multi-type tree structure into coding units (CUs), with aleaf coding unit (CU) defining a region sharing the same prediction mode(e.g. intra or inter). Here, the term “unit” defines a region of animage covering all components; the term “block” is used to define aregion covering a particular component (e.g. luma), and may differ inspatial location when considering the chroma sampling format such as4:2:0.

Partitioning of the Picture into CTUs

FIG. 5 is a schematic diagram illustrating an example of a picturedivided into CTUs in accordance with some implementations of the presentdisclosure.

In VVC, pictures are divided into a sequence of CTUs, and the CTUconcept is the same as that of the HEVC. For a picture that has threesample arrays, a CTU consists of an N×N block of luma samples togetherwith two corresponding blocks of chroma samples. FIG. 5 shows theexample of a picture 500 divided into CTUs 502.

The maximum allowed size of the luma block in a CTU is specified to be128×128 (although the maximum size of the luma transform blocks is64×64).

Partitioning of the CTUs Using a Tree Structure

FIG. 6 is a schematic diagram illustrating multi-type tree splittingmodes in accordance with some implementations of the present disclosure.

In HEVC, a CTU is split into CUs by using a quaternary-tree structuredenoted as coding tree to adapt to various local characteristics. Thedecision whether to code a picture area using inter-picture (temporal)or intra-picture (spatial) prediction is made at the leaf CU level. Eachleaf CU can be further split into one, two or four PUs according to thePU splitting type. Inside one PU, the same prediction process isapplied, and the relevant information is transmitted to the decoder on aPU basis. After obtaining the residual block by applying the predictionprocess based on the PU splitting type, a leaf CU can be partitionedinto transform units (TUs) according to another quaternary-treestructure similar to the coding tree for the CU. One of key feature ofthe HEVC structure is that it has the multiple partition conceptionsincluding CU, PU, and TU.

In VVC, a quadtree with nested multi-type tree using binary and ternarysplits segmentation structure replaces the concepts of multiplepartition unit types, i.e. it removes the separation of the CU, PU andTU concepts except as needed for CUs that have a size too large for themaximum transform length, and supports more flexibility for CU partitionshapes. In the coding tree structure, a CU may have either a square orrectangular shape. A coding tree unit (CTU) is first partitioned by aquaternary tree (i.e., quadtree) structure. Then the quaternary treeleaf nodes can be further partitioned by a multi-type tree structure. Asshown in FIG. 6 , there are four splitting types in multi-type treestructure: vertical binary splitting 602 (SPLIT_BT_VER), horizontalbinary splitting 604 (SPLIT_BT_HOR), vertical ternary splitting 606(SPLIT_TT_VER), and horizontal ternary splitting 608 (SPLIT_TT_HOR). Themulti-type tree leaf nodes are called coding units (CUs), and unless theCU is too large for the maximum transform length, this segmentation isused for prediction and transform processing without any furtherpartitioning. This means that, in most cases, the CU, PU and TU have thesame block size in the quadtree with nested multi-type tree coding blockstructure. The exception occurs when the maximum supported transformlength is smaller than the width or height of the color component of theCU. In VTM1, a CU consists of coding blocks (CBs) of different colorcomponents, e.g., one CU contains one luma CB and two chroma CBs (unlessthe video is monochrome, i.e., having only one color component).

Partitioning CUs into Multiple Prediction Units

In VVC, for each CU partitioned based on the structure illustratedabove, prediction of the block content can be performed either on thewhole CU block or in a sub-block manner explained in the followingparagraphs. The operation unit of such prediction is called predictionunit (or PU).

In the case of intra prediction (or intra-frame prediction), usually thesize of the PU is equal to the size of the CU. In other words, theprediction is performed on the whole CU block. For inter prediction (orinter-frame prediction), the size of the PU can be equal or less thanthe size of the CU. In other words, there are cases where a CU may besplit into multiple PUs for prediction.

Some examples of having the PU size smaller than the CU size include anaffine prediction mode, an Advanced Temporal Level Motion VectorPrediction (ATMVP) mode, and a triangle prediction mode, etc.

Under the affine prediction mode, a CU may be split into multiple 4×4PUs for prediction. Motion vectors can be derived for each 4×4 PU andmotion compensation can be performed accordingly on the 4×4 PU. Underthe ATMVP mode, a CU may be split into one or multiple 8×8 PUs forprediction. Motion vectors are derived for each 8×8 PU and motioncompensation can be performed accordingly on the 8×8 PU. Under thetriangle prediction mode, a CU may be split into two triangular shapeprediction units. Motion vectors are derived for each PU and motioncompensation is performed accordingly. The triangle prediction mode issupported for inter prediction. More details of the triangle predictionmode are illustrated below.

Triangle Prediction Mode

FIG. 7 is a schematic diagram illustrating splitting a CU intotriangular prediction units in accordance with some implementations ofthe present disclosure.

The concept of the triangle prediction mode is to introduce triangularpartitions for motion compensated prediction. As shown in FIG. 7 , a CU702, 704 is split into two triangular prediction units PU₁ and PU₂, ineither the diagonal or the inverse diagonal direction (i.e., eithersplitting from top-left corner to bottom-right corner or splitting fromtop-right corner to bottom-left corner). Each triangular prediction unitin the CU is inter-predicted using its own uni-prediction motion vectorand reference frame index which are derived from a uni-predictioncandidate list. An adaptive weighting process is performed to thediagonal edge after predicting the triangular prediction units. Then,the transform and quantization process are applied to the whole CU. Itis noted that this mode is only applied to skip and merge modes in thecurrent VVC. Although in FIG. 7 , the CU is shown as a square block, thetriangle prediction mode may be applied to non-square (i.e. rectangular)shape CUs as well.

The uni-prediction candidate list may comprise one or more candidates,and each candidate may be a motion vector. Thus, throughout thisdisclosure, the terms “uni-prediction candidate list,” “uni-predictionmotion vector candidate list,” and “uni-prediction merge list” may beused interchangeably.

Uni-Prediction Motion Vector Candidate List

FIG. 8 is a schematic diagram illustrating positions of the neighboringblocks in accordance with some implementations of the presentdisclosure.

In some examples, the uni-prediction motion vector candidate list mayinclude two to five uni-prediction motion vector candidates. In someother examples, other number may also be possible. It is derived fromneighboring blocks. The uni-prediction motion vector candidate list isderived from seven neighboring blocks including five spatial neighboringblocks (1 to 5) and two temporal co-located blocks (6 to 7), as shown inFIG. 8 . The motion vectors of the seven neighboring blocks arecollected into a first merge list. Then, a uni-prediction candidate listis formed based on the first merge list motion vectors according to aspecific order. Based on the order, the uni-prediction motion vectorsfrom the first merge list are put in the uni-prediction motion vectorcandidate list first, followed by reference picture List 0 or L0 motionvector of bi-prediction motion vectors, and then reference picture List1 or L1 motion vector of bi-prediction motion vectors, and then followedby the averaged motion vector of the L0 and L1 motion vectors ofbi-prediction motion vectors. At that point, if the number of candidatesis still less than a target number (which is five in the current VVC),zero motion vectors are added to the list to meet the target number.

A predictor is derived for each of the triangular PUs based on itsmotion vector. It is worth noting that the predictor derived covers alarger area than the actual triangular PU so that there is an overlappedarea of the two predictors along the shared diagonal edge of the twotriangular PUs. An adaptive weighting process is applied to the diagonaledge area between the two predictors to derive a final prediction forthe CU. Two sets of weighting factors are currently used as follows:

-   -   1st weighting factor set: {7/8, 6/8, 4/8, 2/8, 1/8} and {7/8,        4/8, 1/8} are used for the luminance and the chrominance        samples, respectively;    -   2nd weighting factor set: {7/8, 6/8, 5/8, 4/8, 3/8, 2/8, 1/8}        and {6/8, 4/8, 2/8} are used for the luminance and the        chrominance samples, respectively.

Selection of the weighting factor set is based on the comparison of themotion vectors of the two triangular prediction units. Morespecifically, the 2nd weighting factor set is used when the referencepictures of the two triangular prediction units are different from eachother or their motion vector difference is larger than 16 pixels.Otherwise, the 1st weighting factor set is used. FIG. 9 shows an exampleof weighting with a weighting factor set in accordance with someimplementations of the present disclosure. In FIG. 9 , the 1st weightingfactor set is used for the luminance samples 902 and the chrominancesamples 904.

Triangle Prediction Mode Syntax and Signaling

In the current VVC, the triangle prediction mode is signaled using atriangle prediction flag. This flag is signaled when a CU is coded ineither skip mode or merge mode. For a given CU, if the flag has a valueof 1, it means that the corresponding CU is coded using the triangleprediction mode. Otherwise, the CU is coded using a prediction modeother than the triangle prediction mode.

According to the current VVC standard draft, the triangle predictionflag is conditionally signaled in either skip mode or merge mode.Firstly, a triangle prediction tool enable/disable flag is signaled insequence parameter set (or SPS). Only if this flag is true, the triangleprediction flag is signaled at the CU level. Secondly, the triangleprediction tool is only allowed in B slices. Thus, only in a B slice,the triangle prediction flag is signaled at the CU level. Thirdly, thetriangle prediction mode is signaled only for a CU with a size equal orlarger than a certain threshold. If a CU has a size smaller than thatthreshold, the triangle prediction flag is not signaled. Fourthly, thetriangle prediction flag is only signaled for a CU if that CU is notcoded in the sub-block merge mode which includes both the affineprediction mode and the ATMVP mode. In the four cases listed above, ifthe triangle prediction flag is not signaled, it is inferred as 0 at thedecoder side. In some examples, the triangle prediction flag is notexplicitly signaled in the bitstream. Instead, the triangle predictionmode is inferred as enabled when all the other merge related modes aresignaled as disabled.

According to the current VVC standard draft, when the triangleprediction flag is signaled, it is signaled using a CABAC entropy coderwith certain contexts. The contexts are formed based on the triangleprediction flag values from neighboring blocks, as exemplified in FIG.10 which illustrates context derivation for coding the triangleprediction flag in accordance with some implementations of the presentdisclosure.

As shown in FIG. 10 , to code (i.e. either encode or decode) a triangleprediction flag for a current block 1000 (or a current CU), the triangleprediction flags from both the top block 1004 and the left block 1002(or the top and the left CUs) are derived and their values are summedup. This results in three possible contexts corresponding to thefollowing cases:

-   -   1) Both the left block and the top block have a triangle        prediction flag of 0;    -   2) Both the left block and the top block have a triangle        prediction flag of 1;    -   3) Otherwise.

Separate probabilities are maintained for each of the three contexts.Once a context value is determined for a current block, the triangleprediction flag of the current block is coded using the CABACprobability model corresponding to that context value.

If the triangle prediction flag is true, a table index value related tothe triangle prediction mode is further signaled. This index value canbe used to derive the following information:

-   -   1) the triangle prediction partition orientation, i.e. whether        the block is partitioned from the top-left corner to        bottom-right corner, or from the top-right corner to bottom-left        corner; and    -   2) a motion vector in the uni-prediction motion vector candidate        list for each of the two partitions.

More specifically, this is achieved through a table lookup operation,e.g., with a triangle prediction merge list index signaling table asshown in Table 1. In the current VVC reference software VTM, this table,which is an array of size 40 by 3, is defined as follows.

TABLE 1 Triangle prediction merge list index signaling table  g_triangleCombination[40][3] = {  {0, 1, 0}, {1, 0, 1}, {1, 0, 2}, {0,0, 1}, {0, 2, 0},  {1, 0, 3}, {1, 0, 4}, {1, 1, 0}, {0, 3, 0}, {0, 4,0},  {0, 0, 2}, {0, 1, 2}, {1, 1, 2}, {0, 0, 4}, {0, 0, 3},  {0, 1, 3},{0, 1, 4}, {1, 1, 4}, {1, 1, 3}, {1, 2, 1},  {1, 2, 0}, {0, 2, 1}, {0,4, 3}, {1, 3, 0}, {1, 3, 2},  {1, 3, 4}, {1, 4, 0}, {1, 3, 1}, {1, 2,3}, {1, 4, 1},  {0, 4, 1}, {0, 2, 3}, {1, 4, 2}, {0, 3, 2}, {1, 4, 3}, {0, 3, 1}, {0, 2, 4}, {1, 2, 4}, {0, 4, 2}, {0, 3, 4}, };

This 2-dimensional array has 40 rows and 3 columns. The values of thefirst column indicate the partition orientation, with 0 indicating onepartition orientation and 1 indicating the other partition orientation.The values of the second and third column are index values pointing tothe corresponding uni-prediction motion vectors in the uni-predictionmotion vector candidate list, for each of the two partitions, e.g., thetwo triangular partitions, respectively. As illustrated earlier,according to the current VVC standard draft, the uni-prediction motionvector candidate list includes two to five uni-prediction motionvectors. The index values indicate which motion vector in this list isto be used for the corresponding triangular partition.

Under the triangle prediction mode, the table index value signaledranges from 0 to 39. Once the index value is decoded, it can be used tolook up a row of data including three numbers according to the tableabove. These three numbers indicate the triangle prediction partitionorientation and a merge list index value of the uni-prediction mergelist for each of the two motion vectors of the triangular partitions.

In the current VVC standard draft, to extend and/or simplify thetriangle prediction mode, the idea of the triangular PU is extended toother partitions of prediction units, as shown in FIG. 11 whichillustrates examples of PU partitions in accordance with someimplementations of the present disclosure. In addition to the twodiagonal triangular PU partitions, the method or concept used in thetriangle prediction mode may be applied to other PU partitions.Basically, given a type of partition shown in FIG. 11 , twouni-directional predictors are derived for each partition (e.g., PU0 andPU1), and adaptive weighting is applied around the border area of thetwo partitions. For example, the method used for triangular PUs may beapplied to horizontal and/or vertical PUs.

Regular Merge Mode Motion Vector Candidate List

According to the current VVC, under the regular merge mode where a wholeCU is predicted without splitting into more than one PU, the motionvector candidate list or the merge candidate list is constructed using adifferent procedure than that for the triangle prediction mode.

Firstly, spatial motion vector candidates are selected based on motionvectors from neighboring blocks as indicated in FIG. 12 , which is aschematic diagram illustrating positions of spatial merge candidates inaccordance with some implementations of the present disclosure. In thederivation of spatial merge candidates, a maximum of four mergecandidates are selected among candidates that are located in positionsas depicted in FIG. 12 . The order of derivation is A₁→B₁→B₀→A₀→(B₂).The position B2 is considered only when any PU of positions A₁, B₁, B₀,A₀ is not available or is intra coded.

Next, a temporal merge candidate is derived. In the derivation of thetemporal merge candidate, a scaled motion vector is derived based on theco-located PU belonging to the picture which has the smallest PictureOrder Count (POC) difference with the current picture within the givenreference picture list. The reference picture list to be used forderivation of the co-located PU is explicitly signaled in the sliceheader. The scaled motion vector for the temporal merge candidate isobtained as illustrated by the dotted line in FIG. 13 which illustratesmotion vector scaling for the temporal merge candidate in accordancewith some implementations of the present disclosure. The scaled motionvector for the temporal merge candidate is scaled from the motion vectorof the co-located PU col_PU using the POC distances, tb and td, where tbis defined to be the POC difference between the reference picture of thecurrent picture curr_ref and the current picture curr_pic and td isdefined to be the POC difference between the reference picture of theco-located picture col ref and the co-located picture col_pic. Thereference picture index of the temporal merge candidate is set equal tozero. A practical realization of the scaling process is described in theHEVC draft specification. For a B-slice, two motion vectors, one forreference picture List 0 and the other for reference picture List 1, areobtained and combined to make the bi-predictive merge candidate.

FIG. 14 is a schematic diagram illustrating candidate positions for thetemporal merge candidate in accordance with some implementations of thepresent disclosure.

The position of co-located PU is selected between two candidatepositions, C3 and H, as depicted in FIG. 14 . If the PU at position H isnot available, or is intra coded, or is outside of the current CTU,position C3 is used for the derivation of the temporal merge candidate.Otherwise, position H is used for the derivation of the temporal mergecandidate.

After inserting both spatial and temporal motion vectors into the mergecandidate list as described above, history-based merge candidates areadded. The so-called history-based merge candidates include those motionvectors from previously coded CUs, which are maintained in a separatemotion vector list, and managed based on certain rules.

After inserting history-based candidates, if the merge candidate list isnot full, pairwise average motion vector candidates are further addedinto the list. As its name indicates, this type of candidates isconstructed by averaging candidates already in the current list. Morespecifically, based on a certain order, two candidates in the mergecandidate list are taken each time and the average motion vector of thetwo candidates is appended to the current list.

According to the current VVC, the following motion vector paring orderis used to determine which two candidates in the list are taken eachtime to derive an average motion vector candidate:

-   -   {0, 1} →{0, 2} →{1, 2}→{0, 3} →{1, 3} →{2, 3}.

The two values in each pair of braces represent the two index values ofmotion vector candidates in the merge candidate list. Thus, the firstaverage motion vector candidate is generated by averaging motion vectorcandidate 0 and motion vector candidate 1 in the merge candidate list.The second average motion vector candidate is generated by averagingmotion vector candidate 0 and motion vector candidate 2 in the mergecandidate list, and so on.

After inserting pairwise average motion vectors, if the merge candidatelist is still not full, zero motion vectors will be added to make thelist full.

Merge Mode with Motion Vector Difference (MMVD)/Ultimate Motion VectorExpression (UMVE)

Ultimate motion vector expression (UMVE) is adopted in the VVC and hasbeen integrated into the reference software VTM. The UMVE is laterrenamed as merge mode with MVD (MMVD). MMVD is used for either skip modeor merge mode with a proposed motion vector expression method.

MMVD re-uses merge candidates in a same manner as in the VVC. Among themerge candidates, a candidate can be selected, and is further expandedby the proposed motion vector expression method.

MMVD provides a new motion vector expression with simplified signalling.The expression method includes a starting point, a motion magnitude, anda motion direction.

The proposed technique uses a merge candidate list as it is. But onlycandidates which are of default merge type (MRG_TYPE_DEFAULT_N) areconsidered for MMVD's expansion.

A base candidate index (IDX) defines the starting point. The basecandidate index indicates the best candidate among candidates in thetable below.

TABLE 2 Base candidate IDX Base candidate IDX 0 1 2 3 N^(th) MVP 1^(st)MVP 2^(nd) MVP 3^(rd) MVP 4^(th) MVP

If the number of base candidates is equal to 1, the base candidate IDXis not signalled.

A distance index is the motion magnitude information. The distance indexindicates the pre-defined distance from the starting point. Thepre-defined distance is as follows:

TABLE 3 Distance IDX Distance IDX 0 1 2 3 4 5 6 7 Pixel 1/4-pel 1/2-pel1-pel 2-pel 4-pel 8-pel 16-pel 32-pel distance

A direction index represents the direction of the MVD relative to thestarting point. The direction index can represent the four directions asshown in the table below.

TABLE 4 Direction IDX Direction IDX 00 01 10 11 x-axis + − N/A N/Ay-axis N/A N/A + −

A MMVD flag is signalled right after sending a skip flag and merge flag.If the skip and merge flag is true, the MMVD flag is parsed. If the MMVDflag is equal to 1, MMVD syntaxes are parsed; otherwise, an AFFINE flagis parsed. If the AFFINE flag is equal to 1, AFFINE mode is indicated;otherwise, skip/merge index is parsed for VTM's skip/merge mode.

Combined Inter and Intra Prediction (CIIP)

In VTM, when a CU is coded in the merge mode, if the CU contains atleast 64 luma samples (that is, the CU width times the CU height isequal to or larger than 64), an additional flag is signalled to indicatewhether the combined inter/intra prediction (CIIP) mode is applied tothe current CU.

In order to form the CIIP, an intra prediction mode is first derivedfrom two additional syntax elements. Up to four possible intraprediction modes can be used: DC, planar, horizontal, or vertical. Then,the inter prediction and intra prediction signals are derived usingregular intra and inter decoding processes. Finally, weighted averagingof the inter and intra prediction signals is performed to obtain theCIIP prediction.

In the intra prediction mode derivation, up to 4 intra prediction modes,including DC, PLANAR, HORIZONTAL, and VERTICAL modes, can be used topredict the luma component in the CIIP mode. If the CU shape is verywide (that is, its width is more than two times of its height), theHORIZONTAL mode is not allowed. If the CU shape is very narrow (that is,its height is more than two times of its width), the VERTICAL mode isnot allowed. In these cases, only 3 intra prediction modes are allowed.

The CIIP mode uses 3 most probable modes (MPM) for intra prediction. TheCIIP MPM candidate list is formed as follows:

The left and top neighboring blocks are set as A and B, respectively;

The intra prediction modes of block A and block B, denoted as intraModeAand intraModeB, respectively, are derived as follows:

-   -   i. Let X be either A or B;    -   ii. intraModeX is set to DC if 1) block X is not available;        or 2) block X is not predicted using the CIIP mode or the intra        mode; 3) block X is outside of the current CTU;    -   iii. otherwise, intraModeX is set to 1) DC or PLANAR if the        intra prediction mode of block X is DC or PLANAR; or 2) VERTICAL        if the intra prediction mode of block X is a “vertical-like”        angular mode (larger than 34), or 3) HORIZONTAL if the intra        prediction mode of block X is a “horizontal-like” angular mode        (smaller than or equal to 34);

If intraModeA and intraModeB are the same:

-   -   i. If intraModeA is PLANAR or DC, then the three MPMs are set to        {PLANAR, DC, VERTICAL} in that order;    -   ii. Otherwise, the three MPMs are set to {intraModeA, PLANAR,        DC} in that order;

Otherwise (intraModeA and intraModeB are different):

-   -   i. The first two MPMs are set to {intraModeA, intraModeB} in        that order;    -   ii. Uniqueness of PLANAR, DC and VERTICAL is checked in that        order against the first two MPM candidate modes; as soon as a        unique mode is found, it is added to as the third MPM.

If the CU shape is very wide or very narrow as defined above, an MPMflag is inferred to be 1 without signalling. Otherwise, the MPM flag issignalled to indicate if the CIIP intra prediction mode is one of theCIIP MPM candidate modes.

If the MPM flag is 1, an MPM index is further signalled to indicatewhich one of the MPM candidate modes is used in the CIIP intraprediction. Otherwise, if the MPM flag is 0, the intra prediction modeis set to the “missing” mode in the MPM candidate list. For example, ifthe PLANAR mode is not in the MPM candidate list, the PLANAR mode is themissing mode, and the intra prediction mode is set to PLANAR. Since 4possible intra prediction modes are allowed in the CIIP, and the MPMcandidate list contains only 3 intra prediction modes, one of the 4possible modes must be the missing mode.

For the chroma components, the DM mode is always applied withoutadditional signalling; that is, chroma uses the same prediction mode asluma. In this example, the chroma intra prediction mode directly reusesthe intra prediction mode for the luma component. It is thus namedderived mode, or DM mode.

The intra prediction mode of a CIIP-coded CU will be saved and used inthe intra mode coding of the future neighboring CUs.

To combine the inter and intra prediction signals, the inter predictionsignal in the CIIP mode P_inter is derived using the same interprediction process applied to the regular merge mode; and the intraprediction signal P_intra is derived using the CIIP intra predictionmode following the regular intra prediction process. Then, the intra andinter prediction signals are combined using weighted averaging, wherethe weight value depends on the intra prediction mode and where thesample is located in the coding block, as follows:

If the intra prediction mode is the DC or planar mode, or if the blockwidth or height is smaller than 4, equal weights are applied to theintra prediction and the inter prediction signals.

Otherwise, the weights are determined based on the intra prediction mode(either horizontal mode or vertical mode in this case) and the samplelocation in the block. Taking the horizontal prediction mode for example(the weights for the vertical mode are derived similarly but in theorthogonal direction) and denoting W as the width of the block and H asthe height of the block, the coding block is first split into fourequal-area parts, each having a dimension of (W/4)×H. Starting from thepart closest to the intra prediction reference samples and ending at thepart farthest away from the intra prediction reference samples, theweight wt for each of the four parts or regions is set to 6, 5, 3, and2, respectively. The final CIIP prediction signal is derived using thefollowing equation:P _(CIIP)=((8−wt)*P _(inter) +wt*P _(intra))»³.

As previously described, the triangle prediction merge list indexes aresignaled based on a pre-defined Table 1 of size 40×3. The first columnof the table indicates the triangle partition orientation, and thesecond and third columns indicate the index values of the motion vectorcandidates in the uni-prediction motion vector candidate list for eachof the two triangle prediction units of a CU. The table needs to bestored at both the encoder and the decoder ends. In some examples of thepresent disclosure, it is proposed that the size of the table may bereduced.

Separate Signaling of Triangle Partition Orientation and Merge ListIndexes

The current Table 1 defined in VVC for triangle prediction merge listindex signaling may be derived through data training. In this table, thetriangle partition orientation and the merge list indexes are groupedtogether for signaling. Theoretically, it may be unlikely that onetriangle partition orientation always has a higher chance to be usedthan the other triangle partition orientation. Therefore, based on thepresent disclosure, the signaling of the triangle partition orientationmay be decoupled from the merge list index signaling.

More specifically, according to the present disclosure, under thetriangle prediction mode, a separate flag, named a partition orientationflag or a triangle partition orientation flag for example, may besignaled to indicate which of the two triangle partition orientations isused. The two merge list index values indicating selected entries in theuni-prediction merge list or the uni-prediction motion vector candidatelist, one for each triangular partition, are signaled together as atable index value.

For example, the following Table 5, having a dimension of 20×2, may beused for signaling of the merge list index values of the two triangleprediction units. In this case, a table index value ranging from 0 to 19may be signaled to indicate which merge list index values are used forthe two triangular partitions, respectively.

TABLE 2 Simplified triangle prediction merge list index signaling  g_triangleCombination[20][2] = {  {0, 1}, {1, 0}, {0, 2}, {2, 0}, {1,2},  {2, 1}, {0, 3}, {3, 0}, {1, 3}, {3, 1},  {0, 4}, {4, 0}, {2, 3},{3, 2}, {1, 4},  {4, 1}, {2, 4}, {4, 2}, {3, 4}, {4, 3} };

It should be noted that the table above shows just one example. Othercombination order may be used.

It should be also noted that the signaling order of the trianglepartition orientation flag and the table index value (which represents amerge list index combination) may also vary. In other words, thetriangle partition orientation flag may be signaled before the tableindex value, or after the table index value.

According to another example of the present disclosure, the trianglepartition orientation flag may be simply coded with an assumption thatthe flag has equal probabilities (i.e. 50% probability) of having avalue of 0 and having a value of 1. In other words, the flag may bealways coded as CABAC bypass bin.

Accordingly, in some examples, the method includes: partitioning videopictures into a plurality of coding units (CUs), at least one of whichis further portioned into two prediction units (PUs) including at leastone triangular shaped PU with a partitioning orientation in one of: fromtop-left corner to bottom-right corner, and from top-right corner tobottom-left corner; constructing a uni-prediction motion vectorcandidate list; determining whether a current CU is coded as triangleprediction mode according to coded information; signaling a partitionorientation flag indicating the partitioning orientation; and signalingindex values that indicate selected entries in the constructeduni-prediction motion vector candidate list.

Separate Signaling of Merge Index Value and Partition Mapping

To further reduce memory requirement, according to another example ofthe present disclosure, under the triangle prediction mode, anadditional flag, named a mapping flag for example, may be signaled toindicate how the index values are assigned to the two triangularpartitions, or a mapping scheme between the two index values and the twotriangular partitions.

More specifically, the method is based on the fact that some symmetryexists in the data shown in Table 5. For example, considering the firsttwo rows of {0, 1} and {1, 0} in the table, they represent the same twomotion vectors in the uni-prediction merge list being assigned to thetwo triangular partitions, but with different correspondences. {0, 1}indicates that the first motion vector candidate in the merge list isassigned to the first partition and the second motion vector candidatein that merge list is assigned to the second partition. {1, 0} indicatesthat the second motion vector candidate in the merge list is assigned tothe first partition and the first motion vector candidate in that listis assigned to the second partition. According to some examples of thepresent disclosure, these two cases can be both signaled using {0, 1},together with a mapping flag signaled to differentiate between the twocases.

TABLE 3 Further simplified triangle prediction merge list indexsignaling   g_triangleCombination[10][2] = {  {0, 1},{0, 2}, {1, 2}, {0,3}, {1, 3},  {0, 4},{2, 3}, {1, 4}, {2, 4}, {3, 4}, };

Based on the illustrations above, the table may be further simplified asTable 6, having a dimension of 10×2. In this case, a table index valueranging from 0 to 9 is signaled to indicate which merge list indexvalues are used for the two triangular partitions. A mapping flag issignaled together with the table index value. For example, if the tableindex value of 0 is signaled, it indicates {0, 1} which means that thefirst and the second motion vector candidates in the merge list areused. If the mapping flag is 0, it may indicate that the first motionvector candidate in the merge list is assigned to the first partitionand the second motion vector candidate in that list is assigned to thesecond partition. Otherwise, if the mapping flag is 1, it may indicatethat the second motion vector candidate in the merge list is assigned tothe first partition and the first motion vector candidate in that listis assigned to the second partition.

It should be noted that the table above shows just one example. Othercombination order may be used.

It should be also noted that the signaling order of the mapping flag andthe table index value (which represents a merge list index combination)may also vary. In other words, the mapping flag may be signaled beforethe table index value, or after the table index value.

According to another example of the present disclosure, the mapping flagmay be simply coded with an assumption that the flag has equalprobabilities (i.e. 50% probability) of having a value of 0 and having avalue of 1. In other words, the flag may be always coded as CABAC bypassbin.

Sharing the Merge List Index Combination Table with Pairwise AverageMotion Vector

As previously explained, during the regular merge mode motion vectorcandidate list generation process, a motion vector pairing order is usedin generating pairwise average motion vectors. The motion vector pairingorder shown in the example is:

-   -   {0, 1} →{0, 2} →{1, 2} →{0, 3} →{1, 3} →{2, 3}.

According to another example of the present disclosure, only one tablemay be defined and the table may be shared for both a) the regular modepairwise averaging motion vector generation, and b) the triangleprediction mode merge list index signaling.

For example, Table 6 may be further modified as Table 7 below so that itmay be also used for regular mode pairwise averaging motion vectorgeneration.

TABLE 4 Shared index combination   g_triangleCombination[10][2] = {  {0,1}, {0, 2}, {1, 2}, {0, 3}, {1, 3},  {2, 3}, {0, 4}, {1, 4}, {2, 4}, {3,4}, };

It may be noted that the first six entries of Table 7 above areidentical to that of the previously shown motion vector pairing order.Accordingly, the six consecutive entries may be deemed a shared section.

In another aspect as previously described, according to the current VVCstandard draft, the triangle prediction flag is coded based on contextsderived from triangle prediction flag values of the neighboring blocks.However, for actual video content, the correlation among neighboringblocks in terms of selecting the triangle prediction mode may not behigh. Some alternative ways of forming the contexts are proposed incoding the triangle prediction flag.

The triangle prediction mode is usually not used when the CIIP mode isselected in current video coding standards. It is not efficient toalways signal the triangle prediction flag for a current CU regardlessof whether the CIIP mode is selected for the CU or not.

Further, according to the current VVC standard draft, the triangleprediction flag is signaled regardless of whether the MMVD flag is trueor not. If the MMVD mode is not used together with the triangleprediction mode, the signaling could also be made mutually exclusive toimprove coding efficiency.

Coding Triangle Prediction Flag Using Different Contexts

According to some examples of the present disclosure, if the triangleprediction flag needs to be coded (encoded or decoded) for a current CU,contexts may be derived differently from the current VVC standard draftfor CABAC coding.

In some examples of the present disclosure, the contexts are derivedbased on CU area size (i.e. the value of CU height multiplying with thevalue of CU width). In other words, CUs in similar sizes may share asame context.

In some other examples of the present disclosure, the contexts areformed based on CU shape (i.e. the ratio of the CU height to the CUwidth). For example, horizontal orientated rectangular CUs (i.e. the CUwidth is greater than the CU height) may share a first context, andvertically orientated rectangular CUs (i.e. the CU height is greaterthan the CU width) may share a second context, and square CUs (i.e. theCU height is equal to the CU width) may share a third context.

In another example, regardless of horizontal or vertical orientations,rectangular shape CUs with similar ratios of the long side to the shortside share one context, and square shape CUs share another context. Inthis case, multiple contexts may be used with each context correspondingto a different ratio of the CU long side to the CU short side, and/or adifferent range of ratio of the CU long side to the CU short side. Forexample, a first context for CUs having a ratio of the CU long side tothe CU short side greater than 4; a second context for CUs having aratio of the CU long side to the CU short side greater than 2 but lessthan 4; a third context for CUs having a ratio of the CU long side tothe CU short side greater than 1.5 but less than 2; a fourth context forCUs having a ratio of the CU long side to the CU short side greater than1 but less than 1.5; and a fifth context for square CUs. Otherimplementations are also possible.

According to yet another example of the present disclosure, the triangleprediction flag is always coded using one common context, regardless ofCU parameters and/or conditions.

Coding Triangle Prediction Flag Relative to CIIP Flag

According to some examples of the present disclosure, the triangleprediction flag may not be coded for a CU if the CIIP prediction mode isselected for the CU. Thus, only when the CIIP prediction mode is notselected for a CU, the triangle prediction flag may be signaled for theCU. In the case that the triangle prediction flag is not signaled, itmay be inferred as 0 at the decoder side.

Coding Triangle Prediction Flag Relative to MMVD Flag

According to some other examples of the present disclosure, the triangleprediction flag may not be coded for a CU when the MMVD prediction modeis selected for the CU. Thus, only when the MMVD prediction mode is notselected for a CU, the triangle prediction flag may be signaled for theCU. In the case that the triangle prediction flag is not signaled, itmay be inferred as 0 at the decoder side.

Enabling MMVD Prediction on Top of Triangle Prediction Unit

According to some yet further examples of the present disclosure, theMMVD prediction mode may be enabled on top of the triangle predictionunit. In this case, even if the MMVD prediction mode is selected for aCU, the triangle prediction flag may be additionally signaled toindicate that the triangle prediction unit is used. In the case thatboth the MMVD prediction mode and the triangle prediction mode are truefor a CU, MVD related syntaxes under the MMVD mode may be additionallysignaled for each of the two triangle prediction units.

FIG. 15 is a block diagram illustrating an apparatus for video coding inaccordance with some implementations of the present disclosure. Theapparatus 1500 may be a terminal, such as a mobile phone, a tabletcomputer, a digital broadcast terminal, a tablet device, or a personaldigital assistant.

As shown in FIG. 15 , the apparatus 1500 may include one or more of thefollowing components: a processing component 1502, a memory 1504, apower supply component 1506, a multimedia component 1508, an audiocomponent 1510, an input/output (I/O) interface 1512, a sensor component1514, and a communication component 1516.

The processing component 1502 usually controls overall operations of theapparatus 1500, such as operations relating to display, a telephonecall, data communication, a camera operation and a recording operation.The processing component 1502 may include one or more processors 1520for executing instructions to complete all or a part of steps of theabove method. Further, the processing component 1502 may include one ormore modules to facilitate interaction between the processing component1502 and other components. For example, the processing component 1502may include a multimedia module to facilitate the interaction betweenthe multimedia component 1508 and the processing component 1502.

The memory 1504 is configured to store different types of data tosupport operations of the apparatus 1500. Examples of such data includeinstructions, contact data, phonebook data, messages, pictures, videos,and so on for any application or method that operates on the apparatus1500. The memory 1504 may be implemented by any type of volatile ornon-volatile storage devices or a combination thereof, and the memory1504 may be a Static Random Access Memory (SRAM), an ElectricallyErasable Programmable Read-Only Memory (EEPROM), an ErasableProgrammable Read-Only Memory (EPROM), a Programmable Read-Only Memory(PROM), a Read-Only Memory (ROM), a magnetic memory, a flash memory, amagnetic disk or a compact disk.

The power supply component 1506 supplies power for different componentsof the apparatus 1500. The power supply component 1506 may include apower supply management system, one or more power supplies, and othercomponents associated with generating, managing and distributing powerfor the apparatus 1500.

The multimedia component 1508 includes a screen providing an outputinterface between the apparatus 1500 and a user. In some examples, thescreen may include a LCD and a Touch Panel (TP). If the screen includesa touch panel, the screen may be implemented as a touch screen receivingan input signal from a user. The touch panel may include one or moretouch sensors for sensing a touch, a slide and a gesture on the touchpanel. The touch sensor may not only sense a boundary of a touching orsliding actions, but also detect duration and pressure related to thetouching or sliding operation. In some examples, the multimediacomponent 1508 may include a front camera and/or a rear camera. When theapparatus 1500 is in an operation mode, such as a shooting mode or avideo mode, the front camera and/or the rear camera may receive externalmultimedia data.

The audio component 1510 is configured to output and/or input an audiosignal. For example, the audio component 1510 includes a microphone(MIC). When the apparatus 1500 is in an operating mode, such as a callmode, a recording mode and a voice recognition mode, the microphone isconfigured to receive an external audio signal. The received audiosignal may be further stored in the memory 1504 or sent via thecommunication component 1516. In some examples, the audio component 1510further includes a speaker for outputting an audio signal.

The I/O interface 1512 provides an interface between the processingcomponent 1502 and a peripheral interface module. The above peripheralinterface module may be a keyboard, a click wheel, a button, or thelike. These buttons may include but not limited to, a home button, avolume button, a start button and a lock button.

The sensor component 1514 includes one or more sensors for providing astate assessment in different aspects for the apparatus 1500. Forexample, the sensor component 1514 may detect an on/off state of theapparatus 1500 and relative locations of components. For example, thecomponents are a display and a keypad of the apparatus 1500. The sensorcomponent 1514 may also detect a position change of the apparatus 1500or a component of the apparatus 1500, presence or absence of a contactof a user on the apparatus 1500, an orientation oracceleration/deceleration of the apparatus 1500, and a temperaturechange of apparatus 1500. The sensor component 1514 may include aproximity sensor configured to detect presence of a nearby objectwithout any physical touch. The sensor component 1514 may furtherinclude an optical sensor, such as a CMOS or CCD image sensor used in animaging application. In some examples, the sensor component 1514 mayfurther include an acceleration sensor, a gyroscope sensor, a magneticsensor, a pressure sensor, or a temperature sensor.

The communication component 1516 is configured to facilitate wired orwireless communication between the apparatus 1500 and other devices. Theapparatus 1500 may access a wireless network based on a communicationstandard, such as WiFi, 4G, or a combination thereof. In an example, thecommunication component 1516 receives a broadcast signal or broadcastrelated information from an external broadcast management system via abroadcast channel. In an example, the communication component 1516 mayfurther include a Near Field Communication (NFC) module for promotingshort-range communication. For example, the NFC module may beimplemented based on Radio Frequency Identification (RFID) technology,infrared data association (IrDA) technology, Ultra-Wide Band (UWB)technology, Bluetooth (BT) technology and other technology.

In an example, the apparatus 1500 may be implemented by one or more ofApplication Specific Integrated Circuits (ASIC), Digital SignalProcessors (DSP), Digital Signal Processing Devices (DSPD), ProgrammableLogic Devices (PLD), Field Programmable Gate Arrays (FPGA), controllers,microcontrollers, microprocessors or other electronic elements toperform the above method.

A non-transitory computer readable storage medium may be, for example, aHard Disk Drive (HDD), a Solid-State Drive (SSD), Flash memory, a HybridDrive or Solid-State Hybrid Drive (SSHD), a Read-Only Memory (ROM), aCompact Disc Read-Only Memory (CD-ROM), a magnetic tape, a floppy diskand etc.

FIG. 16 is a flowchart illustrating an exemplary process of video codingfor motion compensated prediction using triangular prediction inaccordance with some implementations of the present disclosure.

In step 1602, the processor 1520 partitions video pictures into aplurality of coding units (CUs), at least one of which is furtherportioned into two prediction units (PUs) including at least onetriangular shaped PU with a partitioning orientation in one of: fromtop-left corner to bottom-right corner 702, and from top-right corner tobottom-left corner 704. In step 1604, the processor 1520 constructs auni-prediction motion vector candidate list. In step 1606, the processor1520 determines whether a current CU is coded as triangle predictionmode according to coded information. In step 1608, the processor 1520signals a partition orientation flag indicating the partitioningorientation. In step 1610, the processor 1520 signals index values thatindicate selected entries in the constructed uni-prediction motionvector candidate list.

FIG. 17 is a flowchart illustrating an exemplary process of video codingfor triangle prediction flag in accordance with some implementations ofthe present disclosure.

In step 1702, the processor 1520 partitions video pictures into aplurality of coding units (CUs), at least one of which is furtherportioned into two prediction units (PUs) including at least onetriangular shaped PU.

In step 1704, the processor 1520 determines whether a triangleprediction flag, which indicates triangle prediction mode, is to bedecoded for a current CU.

In step 1706, the processor 1520 derives the triangle prediction flagusing CABAC upon determining that the triangle prediction flag is to bedecoded for the current CU.

Contexts of the CABAC for the triangle prediction flag are derived basedon at least one selected from a group consisting of: a CU area size; aCU shape; and a common context.

In some examples, there is provided an apparatus for video coding. Theapparatus includes a processor 1520; and a memory 1504 configured tostore instructions executable by the processor; where the processor,upon execution of the instructions, is configured to perform a method asillustrated in FIG. 16 , or a method as illustrated in FIG. 17 .

In some other examples, there is provided a non-transitory computerreadable storage medium 1504, having instructions stored therein. Whenthe instructions are executed by a processor 1520, the instructionscause the processor to perform a method as illustrated in FIG. 16 , or amethod as illustrated in FIG. 17 .

The description of the present disclosure has been presented forpurposes of illustration, and is not intended to be exhaustive orlimited to the present disclosure. Many modifications, variations, andalternative implementations will be apparent to those of ordinary skillin the art having the benefit of the teachings presented in theforegoing descriptions and the associated drawings.

The examples were chosen and described in order to explain theprinciples of the disclosure, and to enable others skilled in the art tounderstand the disclosure for various implementations and to bestutilize the underlying principles and various implementations withvarious modifications as are suited to the particular use contemplated.Therefore, it is to be understood that the scope of the disclosure isnot to be limited to the specific examples of the implementationsdisclosed and that modifications and other implementations are intendedto be included within the scope of the present disclosure.

What is claimed is:
 1. A method for video decoding, comprising:obtaining video pictures, wherein the video pictures are partitionedinto a plurality of coding units (CUs), at least one of which is furtherpartitioned into two prediction units (PUs); and in response todetermining that a current CU is coded in an inter prediction mode,determining whether the current CU is coded in a triangle predictionmode according to coded information; and in response to determining thatthe current CU is coded in the triangle prediction mode, obtaining apartition orientation syntax element indicating a partitioningorientation, and obtaining index values corresponding to uni-predictionmotion vector candidates; wherein determining whether the current CU iscoded in the triangle prediction mode according to the coded informationcomprises: in response to determining that a triangle prediction toolenable/disable flag signaled in sequence parameter set (SPS) is false,inferring that the current CU is not coded in the triangle predictionmode.
 2. The method of claim 1, wherein the uni-prediction motion vectorcandidates comprises two uni-prediction motion vector candidates.
 3. Themethod of claim 1, wherein the partition orientation syntax element iscoded as Context-based Adaptive Binary Arithmetic Coding (CABAC) bypassbin.
 4. The method of claim 1, wherein determining whether the currentCU is coded in the triangle prediction mode according to the codedinformation comprises: determining whether a triangle prediction flag,which indicates triangle prediction mode, is to be 0 or 1 for thecurrent CU, according to the coded information.
 5. The method of claim1, further comprising: inter-predicting each PU using its ownuni-prediction motion vector and reference frame index.
 6. The method ofclaim 1, wherein determining whether the current CU is coded in thetriangle prediction mode according to the coded information furthercomprises: in response to determining that all other merge related modesare signaled as disabled according to the coded information and thetriangle prediction tool enable/disable flag is true, inferring that thecurrent CU is coded in the triangle prediction mode.
 7. The method ofclaim 6, wherein the all other merge related modes comprise CombinedInter and Intra Prediction (CIIP) mode, sub-block merge mode and MergeMode with Motion Vector Difference (MMVD) mode.
 8. An apparatus forvideo decoding, comprising: a processor; and a memory configured tostore instructions executable by the processor; wherein the processor,upon execution of the instructions, is configured to: obtain videopictures, wherein the video pictures are partitioned into a plurality ofcoding units (CUs), at least one of which is further partitioned intotwo prediction units (PUs); and in response to determining that acurrent CU is coded in an inter prediction mode, determine whether thecurrent CU is coded in a triangle prediction mode according to codedinformation; and in response to determining that the current CU is codedin the triangle prediction mode, obtain a partition orientation syntaxelement indicating a partitioning orientation, and obtain index valuescorresponding to uni-prediction motion vector candidates; wherein theprocessor is configured to determine whether the current CU is coded inthe triangle prediction mode according to the coded informationcomprises: in response to determining that a triangle prediction toolenable/disable flag signaled in sequence parameter set (SPS) is false,inferring that the current CU is not coded in the triangle predictionmode.
 9. The apparatus of claim 8, wherein the uni-prediction motionvector candidates comprises two uni-prediction motion vector candidates.10. The apparatus of claim 8, wherein the partition orientation syntaxelement is coded as Context-based Adaptive Binary Arithmetic Coding(CABAC) bypass bin.
 11. The apparatus of claim 8, wherein the processoris configured to determine whether the current CU is coded in thetriangle prediction mode according to the coded information comprising:determine whether a triangle prediction flag, which indicates triangleprediction mode, is to be 0 or 1 for a current CU, according to thecoded information.
 12. The apparatus of claim 8, wherein the processoris further configured to: inter-predict each PU using its ownuni-prediction motion vector and reference frame index.
 13. Theapparatus of claim 8, wherein the processor is configured to determinewhether the current CU is coded in the triangle prediction modeaccording to the coded information further comprising: in response todetermining that all other merge related modes are signaled as disabledaccording to the coded information and the triangle prediction toolenable/disable flag is true, inferring that the current CU is coded inthe triangle prediction mode.
 14. The apparatus of claim 13, wherein theall other merge related modes comprise Combined Inter and IntraPrediction (CIIP) mode, sub-block merge mode and Merge Mode with MotionVector Difference (MMVD) mode.
 15. A non-transitory computer readablestorage medium storing a plurality of programs for execution by acomputing device having one or more processors, wherein the plurality ofprograms, when executed by the one or more processors, cause thecomputing device to perform operations comprising: obtaining videopictures, wherein the video pictures are partitioned into a plurality ofcoding units (CUs), at least one of which is further partitioned intotwo prediction units (PUs); in response to determining that a current CUis coded in an inter prediction mode, determining whether the current CUis coded as triangle prediction mode according to coded information; andin response to determining that the current CU is coded in the triangleprediction mode, obtaining a partition orientation syntax elementindicating a partitioning orientation, and obtaining index valuescorresponding to uni-prediction motion vector candidates; whereindetermining whether the current CU is coded in the triangle predictionmode according to the coded information comprises: in response todetermining that a triangle prediction tool enable/disable flag signaledin sequence parameter set (SPS) is false, inferring that the current CUis not coded in the triangle prediction mode.